Revisiting External Validity: Concerns about Trolley Problems and Other Sacrificial Dilemmas in Moral Psychology

Sacrificial dilemmas, especially trolley problems, have rapidly become the most recognizable scientific exemplars of moral situations; they are now a familiar part of the psychological literature and are featured prominently in textbooks and the popular press. We are concerned that studies of sacrif icial dilemmas may lack experimental, mundane, and psychological realism and therefore suffer from low external validity. Our apprehensions stem from three observations about trolley problems and other similar sacrif icial dilemmas: (i) they are amusing rather than sobering, (ii) they are unrealistic and unrepresentative of the moral situations people encounter in the real world, and (iii) they do not elicit the same psychological processes as other moral situations. We believe it would be prudent to use more externally valid stimuli when testing descriptive theories that aim to provide comprehensive accounts of moral judgment and behavior. Research on morality has experienced a major resurgence over the past decade. A shift away from rationalist theories that dominated the literature for many years created new theoretical space, prompted new questions, and called for new empirical methods. New stimuli created for laboratory studies have spurred research activity and led to many contributions to our understanding of morality. However, we believe it is now important to revisit the methodological principle of external validity.We question whether behavioral scientists who study morality should be concerned that they have become desensitized to potential limitations of stimuli that have risen in prominence over the past several years. To the extent that researchers seek to develop general theories of morality, their study stimuli must engage the same psychological processes that operate in everyday situations (Aronson, Wilson, & Brewer, 1998; Mook, 1983). The scholarly literature on moral judgment increasingly features studies that examine people’s reactions to “sacrificial dilemmas” (Bartels & Pizarro, 2011), or brief scenarios where the only way to prevent a calamity from affecting a group of people would be to harm someone else or some smaller group. The trade-off in sacrificial dilemmas is not problematic in and of itself. Researchers can learn a great deal from the way people approach tough choices that put different moral considerations in conf lict. Our concern, however, is that many sacrificial dilemmas are set in fanciful, sometimes absurd, contexts, and these artificial settings may affect the way people approach the situation and decide what to do. Moral psychology has developed a sophisticated understanding of how people respond to sacrificial dilemmas (Bartels, Bauman, Cushman, Pizarro, & McGraw, in press; Waldmann, Nagel, & Wiegmann, 2012), but we worry that the judgment and decision-making processes people use in these unusual situations © 2014 John Wiley & Sons Ltd External Validity in Moral Psychology 537 may not accurately ref lect moral functioning in a broader set of situations. To be clear, our focus in the current paper is on aspects of commonly used sacrificial dilemmas that make them seem frivolous and different from more realistic moral situations; we find little fault with studying moral dilemmas per se. External validity refers to how well the results of a given study generalize and explain a range of other situations (Campbell, 1957). We contend that the results of experiments that examine people’s responses to artificial sacrificial dilemmas may suffer from low external validity because artificial sacrificial dilemmas often lack experimental, mundane, and psychological realism (Aronson et al., 1998). Experimental realism entails how well the situation meaningfully engages participants and causes them to take the study seriously. Mundane realism refers to how likely it is that the events in a study resemble those that participants confront in their everyday lives. Psychological realism involves whether the same mental processes operate during an experiment and real-world analogues. We suspect that many—and especially the most popular—sacrificial dilemmas score relatively low on all three types of realism, which reduces the extent to which people’s choices about the dilemmas can inform general theories of morality. In the absence of external validity, researchers may collectively be building a science of how people respond to a select set of stimuli that capture only a narrow and perhaps distorted view of moral phenomena rather than generating a comprehensive theory of how people make moral judgments across the full range of moral situations they encounter in their daily lives. To illustrate our concerns about the artificial settings of many sacrificial dilemmas, we examine trolley problems. Trolley problems are the most prominent examples of sacrificial dilemmas. They have been used extensively in experiments, and they acted as the catalyst that brought sacrificial dilemmas into mainstreammoral psychology. We first explain the origin of trolley problems and other sacrificial dilemmas to contrast the purposes for which they were originally conceived by philosophers with how they are currently used by psychologists. We next discuss why psychologists and philosophers generally use different methods and call attention to ways in which experiments that use artificial sacrificial dilemmas may not be externally valid. We also present three observations about trolley problems that illustrate in concrete terms why we are concerned about external validity. Finally, we conclude by calling for researchers to be mindful of external validity when choosing stimulus materials. Before proceeding, we wish to state explicitly that we are not suggesting that researchers completely abandon all sacrificial dilemmas or disregard theories that have been strongly inf luenced by them.We believe that sacrificial dilemmas can be a legitimate source of data, provided that researchers (i) recognize the limitations of unrealistic stimuli and (ii) do not rely on them exclusively. However, the popularity of some sensational examples of sacrificial dilemmas appears to have drawn attention away from external validity by somehow blurring the line between rhetorical devices and scientific stimuli. Moreover, over-reliance on any one class of stimuli can lead to common method variance that can cause the observed relationship between variables to differ from their natural association across a wider range of situations (Campbell & Fiske, 1959). Therefore, we believe it is important to consider the effects of our collective methodological choices.